Math Article Classification

Article Classifier for Mathematical Techniques¶

Given the goal to classify research-level math articles based on the mathematical techniques used, we've discussed several strategies and considerations.

Strategies:¶

Start Simple: Begin with simpler models like Naive Bayes or Logistic Regression. They're interpretable and can work well with small datasets.
Feature Engineering: Using TF-IDF can help highlight important terms in the articles. Consider using bigrams or trigrams to capture more context.
Transfer Learning: Pre-trained models like BERT can be fine-tuned on a small dataset to capture the context and semantics of sentences.
Data Augmentation: Techniques like back translation can increase the size of the training data.
Active Learning: Iteratively train a model, use it to predict unlabeled data, label instances where the model is uncertain, and add them to the training set.
Regularization: Techniques like dropout or L1/L2 regularization can help prevent overfitting, especially with small datasets.
Evaluation: Use techniques like cross-validation for robust evaluation.
Iterative Process: Model building is often iterative. As more data is labeled and the model is refined, its performance should improve.

Current Pipeline:¶

Data Preparation: We assume a DataFrame df with columns 'text' (article content) and 'label' (mathematical technique label).
Text Preprocessing: Clean the text and use TF-IDF to convert articles into a matrix of features.
Model Building: We've started with a Logistic Regression model.
Evaluation: Split the data into training and test sets and evaluate the model's performance on the test set.

Next Steps:¶

Once we have a baseline performance, we can refine the model, explore other models, and potentially leverage the arxiv package to fetch more articles for testing and further training.

from sklearn.naive_bayes import MultinomialNB

# Building a Multinomial Naive Bayes model
mnb_clf = MultinomialNB()
mnb_clf.fit(X_train_tfidf, y_train)

# Predicting on the test set
mnb_y_pred = mnb_clf.predict(X_test_tfidf)

# Evaluating the model
print('Accuracy (Multinomial Naive Bayes):', accuracy_score(y_test, mnb_y_pred))
print('\nClassification Report (Multinomial Naive Bayes):\n', classification_report(y_test, mnb_y_pred))

Models for Text Classification¶

1. Multinomial Naive Bayes (MNB)¶

How it works:¶

Bayes' Theorem: MNB is based on Bayes' theorem, which relates the conditional and marginal probabilities of two random events. It provides a way to calculate the probability of a piece of data belonging to a particular category, given our prior knowledge.
Feature Independence: MNB assumes that the features (words in our case) are conditionally independent given the class. This means that the presence of a particular word is independent of the presence of any other word, given the class label.
Multinomial Distribution: The multinomial distribution describes the probability of observing counts among multiple categories. In the context of text classification, it represents the occurrence of words in the document.

Python Library:¶

Scikit-learn: The MultinomialNB class in the sklearn.naive_bayes module provides the implementation for Multinomial Naive Bayes.

2. Logistic Regression¶

How it works:¶

Linear Model: At its core, logistic regression is a linear model that predicts the probability that a given instance belongs to a particular category.
Sigmoid Function: The linear model's output is transformed using the sigmoid function to produce a value between 0 and 1, representing the probability.
Binary Classification: By default, logistic regression is used for binary classification. For multi-class classification, techniques like 'One-vs-Rest' are used.

Python Library:¶

Scikit-learn: The LogisticRegression class in the sklearn.linear_model module provides the implementation for Logistic Regression.

3. Long Short-Term Memory (LSTM)¶

How it works:¶

Recurrent Neural Network (RNN): LSTM is a type of RNN, which is designed to recognize patterns over time and sequences (like sentences).
Memory Cells: LSTM has memory cells that can maintain information in memory for long periods. This allows it to capture long-term dependencies in the data.
Gates: LSTMs have three gates (input, forget, and output) that regulate the flow of information into, out of, and within the memory cell.

Python Library:¶

Keras: The LSTM class in the keras.layers module provides the implementation for LSTM. Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.

4. Convolutional Neural Network (CNN)¶

How it works:¶

Convolutional Layers: CNNs have convolutional layers that apply convolution operations to the input data. These layers can extract local patterns or features from the data.
Pooling Layers: These layers reduce the spatial dimensions of the data while retaining the most important information.
Fully Connected Layers: After the convolutional and pooling layers, the data is flattened and passed through one or more fully connected layers for classification.
Text Classification: While CNNs are traditionally used for image data, they can also be used for text data by treating text as a one-dimensional "image".

Python Library:¶

Keras: The Conv1D and MaxPooling1D classes in the keras.layers module provide the implementation for one-dimensional convolutions and pooling operations suitable for text data.

These models, combined with the right preprocessing and feature engineering techniques, can be powerful tools for text classification tasks like ours. The choice of model often depends on the nature of the data, the amount of labeled data available, and the specific requirements of the task.

Benefits of Starting Simple for Our Task¶

When we're aiming to classify mathematical articles based on the techniques used therein, starting with simpler models and techniques can offer several advantages specific to our task:

1. Understanding Mathematical Language¶

Mathematical articles have a unique language structure, often interspersed with equations, theorems, and proofs. Simple models can help us quickly identify common patterns, terminologies, and structures that are prevalent in such articles.

2. Identifying Key Features¶

With simpler models like Logistic Regression, we can determine which words or n-grams are most influential in classifying articles. This can provide insights into which mathematical techniques or topics are most distinct and recognizable.

3. Quick Feedback Loop¶

Given the subtlety of distinguishing between advanced mathematical techniques, it's beneficial to have a model that can quickly provide results. This allows for rapid iterations based on feedback, helping refine the classification criteria and labels.

4. Foundation for Data Collection¶

As we progress, we might want to collect more labeled data. Insights from simpler models can guide the selection of articles that need labeling, especially if we adopt an active learning approach.

5. Preprocessing and Data Structures¶

The tokenization, TF-IDF representation, and other preprocessing steps we develop initially can be reused for more advanced models. Additionally, the data structures (like the document-term matrix) and any additional features we engineer can be directly fed into more complex models.

6. Transition to Advanced Techniques¶

Once we have a baseline and understand the nuances of our task, we can explore more advanced techniques like neural embeddings, attention mechanisms, or transformer models. The initial work ensures that we make this transition with a clear understanding of our data and objectives.

7. Resource Efficiency¶

Mathematical articles can be lengthy, and processing them can be resource-intensive. Starting with simpler models ensures that we can work efficiently, even with limited computational resources.

In essence, while our ultimate goal might be to develop a sophisticated classifier that can discern intricate mathematical techniques, starting simple provides a solid foundation. It allows us to understand the landscape, make informed decisions, and set the stage for more advanced explorations.

Specific Insights and Data Structures from Starting Simple¶

Document-Term Matrix (DTM)¶

The Document-Term Matrix (DTM) is a foundational structure in text analysis. Here's a more detailed breakdown:

Values in DTM: In a DTM derived from TF-IDF, the values represent the weighted frequency of terms. The weight is higher if a term appears frequently in a document but not in many documents across the corpus. This helps in emphasizing terms that are unique to specific documents.
Usage in Models:
- Multinomial Naive Bayes: Uses the frequency of terms in the DTM to compute the likelihood of a document belonging to a particular class.
- Logistic Regression: Uses the weighted terms in the DTM as features to predict the probability of a document belonging to a particular class.
- Support Vector Machines (SVM): Can use the DTM to find the hyperplane that best separates different classes in the high-dimensional space of terms.
- Random Forests: Can use the DTM to build multiple decision trees, where decisions are made based on term frequencies.

Vocabulary List¶

The vocabulary list is essentially a list of all unique terms identified during tokenization.

Size: The size of the vocabulary list can give a numeric insight into the diversity of terms in the corpus. For instance, a vocabulary size of 10,000 terms indicates a diverse set of articles, while a size of 500 might suggest a more focused or limited corpus.

Model Coefficients/Parameters¶

For models like Logistic Regression:

Coefficient Values: Each term in the DTM will have an associated coefficient. A positive coefficient indicates that the presence of that term increases the likelihood of the document belonging to a particular class, while a negative coefficient indicates the opposite. The magnitude of the coefficient indicates the strength of this relationship. For instance, a coefficient of 2.5 for the term 'differential equation' might suggest that articles with this term are likely to be about differential equations.

Embeddings (Future Consideration)¶

If we use word embeddings:

Vector Values: Each word will be represented by a dense vector, often with hundreds of dimensions. The values in this vector capture the semantic meaning of the word. For instance, the cosine similarity between the vectors for 'integral' and 'derivative' might be 0.85, indicating they are semantically close in the context of mathematical articles.

In essence, from our initial efforts, we'll have specific data structures like the DTM and vocabulary list, numeric insights like vocabulary size and model coefficients, and potential future values like embedding vectors. Each of these provides concrete, actionable information that can guide our subsequent efforts.

Text Classification Projects Found Online¶

Exploring the web, several projects and articles related to text classification were identified. Let's delve into some of these resources to understand the tools and methodologies they employed:

17 Best Text Classification Datasets for Machine Learning: This article might provide datasets that are commonly used in text classification tasks. Understanding the datasets can give insights into the challenges and nuances of different classification problems.
Machine Learning Projects on Text Classification: This resource could detail specific projects on text classification, including the tools, algorithms, and methodologies employed.
Understanding text classification in NLP with Movie Review Example: While focused on movie reviews, this article from Analytics Vidhya might provide a hands-on approach to text classification, detailing the tools and steps involved.
Machine Learning NLP Text Classification Algorithms and Models: This article could delve into specific algorithms and models used in text classification, providing insights into their strengths, weaknesses, and use cases.

Making Connections to Our Project¶

By exploring these resources, we aim to:

Identify common tools and libraries used in text classification. This can include tokenization libraries, machine learning frameworks, and evaluation tools.
Understand the typical workflow of a text classification project. This can guide our project structure, from data preprocessing to model evaluation.
Learn about challenges and pitfalls faced in similar projects. This can help us anticipate and mitigate potential issues in our project.
Gain insights into best practices in text classification. This can include techniques for handling imbalanced datasets, strategies for hyperparameter tuning, and methods for interpretability.

By standing on the shoulders of those who have tackled similar challenges, we can leverage their experiences and insights to enhance the efficiency and effectiveness of our project.

Comparing and Contrasting with an External Text Classification Project¶

I explored a machine learning project on text classification from the website 'thecleverprogrammer'. Let's delve into one of the projects mentioned in the article and compare it with our goals.

Project: Language Detection¶

Goal of the Project: The objective is to classify texts into different languages. The dataset for this project contains sentences as features and the names of the languages as labels. The aim is to prepare a machine learning model that can classify languages in real-time.

Methodology:

Dataset: Contains sentences from various languages. Each sentence is labeled with the name of the language.
Features: Sentences in different languages.
Labels: Names of the languages.
Model: The specific model used is not mentioned in the article, but typically, text classification tasks for language detection might use models like Naive Bayes, Logistic Regression, or even deep learning models like RNNs.
Evaluation: While not explicitly mentioned, projects of this nature typically use metrics like accuracy, F1-score, or confusion matrices to evaluate the performance of the model.

Comparing with Our Project¶

Similarities:

Both projects involve text classification.
Both require feature extraction from text data.
Both aim to use machine learning models to make real-time predictions.

Differences:

Goal: Our project aims to classify mathematical articles based on the techniques used, whereas the external project focuses on language detection.
Complexity: Classifying mathematical techniques might be more nuanced and complex compared to language detection, which often has clear distinctions between classes.
Dataset: Our dataset would contain mathematical articles or excerpts, while the external project uses sentences from different languages.
Labels: Our labels would be different mathematical techniques, whereas the external project has language names as labels.

Takeaways¶

While the specific goal and dataset differ, the foundational steps of text classification remain consistent. We can draw inspiration from the external project in terms of data preprocessing, feature extraction, model selection, and evaluation. Understanding how similar projects approach text classification can provide valuable insights and guide our methodology.

Handling Mathematical Symbols in Text Classification¶

Mathematical articles often contain a plethora of symbols, each carrying significant meaning. When classifying such articles, it's imperative to effectively handle these symbols to ensure accurate classification. Here are some strategies and considerations for addressing the challenge of mathematical symbols:

Strategies for Handling Mathematical Symbols:¶

Symbol-to-Text Conversion: One approach is to convert mathematical symbols into their textual descriptions. For example, converting the symbol "∫" to "integral" allows models to process these as standard words.
Embeddings for Symbols: Similar to word embeddings, mathematical symbols can be represented using embeddings. Custom embeddings can be trained on the dataset if pre-trained embeddings are unavailable.
Tokenization: By treating mathematical symbols as unique tokens, models can recognize and learn the context of these symbols.
Specialized Libraries: Libraries such as SymPy can parse and standardize mathematical expressions, aiding in data normalization.
Contextual Models: Symbols often derive meaning from their context. Models that capture context, like LSTM or Transformer-based architectures, can be particularly effective.
Data Augmentation: Enhancing the dataset with variations where symbols are interchanged with their textual descriptions can bolster model robustness.
Feature Engineering: For symbols of specific interest, features indicating their presence or absence can be manually crafted.
Leveraging LaTeX: If articles are in LaTeX format, the textual LaTeX commands can be used to discern mathematical content.

Challenges and Solutions:¶

Ambiguity: Symbols can have varied meanings based on context. Contextual models can help discern the intended meaning.
Variability: Different authors might use diverse notations for the same concept. Data normalization and domain-specific knowledge can address this.
Complex Expressions: Nested and intricate mathematical expressions require meticulous parsing. Specialized libraries and domain expertise can be invaluable here.

In conclusion, while mathematical symbols pose challenges in text classification, a combination of the above strategies, coupled with domain knowledge, can lead to effective and accurate classification. As mathematical articles are rich in symbolic content, addressing this aspect is paramount for the success of the classification project.

import PyPDF2

def extract_text_from_pdf(pdf_path):
    with open(pdf_path, 'rb') as file:
        reader = PyPDF2.PdfReader(file)
        text = ''
        for page_num in range(len(reader.pages)):
            text += reader.pages[page_num].extract_text()
    return text

# Extract text from the provided PDFs
paper1_text = extract_text_from_pdf('pypaper1.pdf')
paper2_text = extract_text_from_pdf('pypaper2.pdf')

print(paper1_text[:500])  # Print the first 500 characters of paper1 for verification

A LARGE PAIRWISE FAR FAMILY OF ARONSZAJN TREES
JOHN KRUEGER
Abstract. We construct a large family of normal -complete R-embeddable
non-special +-Aronszajn trees which have no club-isomorphic subtrees using
an instance of the proxy principle of Brodsky-Rinot [5].
Two trees of the same height are said to be club isomorphic if there exists a
club subset of their height and an isomorphism between the trees restricted to
that club. Abraham-Shelah [1] proved a number of essential results about club

Extracting Text from PDFs using PyPDF and PyMuPDF¶

PyPDF Library¶

PyPDF is a Python library that can be used for various PDF-related tasks, including text extraction, generating, decrypting, and merging PDF files. Here's a brief overview of how to extract text from a PDF using PyPDF:

Installation:
```
pip install PyPDF2
```

Usage:

from PyPDF2 import PdfReader
reader = PdfReader('example.pdf')
page = reader.pages[0]
text = page.extract_text()
print(text)

PyMuPDF Library¶

PyMuPDF is another Python library that supports various file formats, including PDF. Here's how to extract text from a PDF using PyMuPDF:

Installation:
```
pip install PyMuPDF==1.16.14
```

Usage:

import fitz
doc = fitz.open('sample.pdf')
text = ""
for page in doc:
    text += page.get_text()
print(text)

Both libraries provide straightforward methods for extracting text from PDFs. The choice between them depends on the specific requirements and preferences of the user. For more details and examples, you can refer to the GeeksforGeeks article.

import gensim
from gensim import corpora

import nltk
nltk.download('stopwords')
nltk.download('punkt')
nltk.download('wordnet')
from nltk.corpus import stopwords  # <-- Add this line

from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
import string


# Preprocess the text
def preprocess(text):
    stop_words = set(stopwords.words('english'))
    tokens = word_tokenize(text)
    tokens = [word for word in tokens if word not in stop_words]
    tokens = [word for word in tokens if word not in string.punctuation]
    lemmatizer = WordNetLemmatizer()
    tokens = [lemmatizer.lemmatize(token) for token in tokens]
    return tokens

# Preprocess the papers
paper1_tokens = preprocess(paper1_text)
paper2_tokens = preprocess(paper2_text)

# Create a dictionary and corpus for LDA
dictionary = corpora.Dictionary([paper1_tokens, paper2_tokens])
corpus = [dictionary.doc2bow(paper) for paper in [paper1_tokens, paper2_tokens]]

# Apply LDA
lda_model = gensim.models.LdaModel(corpus, num_topics=2, id2word=dictionary, passes=15)
topics = lda_model.print_topics(num_words=5)
topics

[(0, '0.020*"\x14" + 0.019*"x" + 0.016*"\x0e" + 0.013*"tree" + 0.010*"1"'),
 (1, '0.017*"tree" + 0.013*"V" + 0.011*"branch" + 0.011*"1" + 0.009*"2"')]

# Enhanced Preprocessing

def enhanced_preprocess(text):
    # Remove special characters often found in PDFs
    text = text.replace('\x14', '').replace('\x0e', '')

    stop_words = set(stopwords.words('english'))
    tokens = word_tokenize(text)
    tokens = [word for word in tokens if word not in stop_words]
    tokens = [word for word in tokens if word not in string.punctuation]
    lemmatizer = WordNetLemmatizer()
    tokens = [lemmatizer.lemmatize(token) for token in tokens]

    # Remove single characters and numbers
    tokens = [word for word in tokens if len(word) > 1 and not word.isdigit()]
    return tokens

# Preprocess the papers using the enhanced preprocessing function
paper1_tokens_enhanced = enhanced_preprocess(paper1_text)
paper2_tokens_enhanced = enhanced_preprocess(paper2_text)

# Create a dictionary and corpus for LDA using the enhanced tokens
dictionary_enhanced = corpora.Dictionary([paper1_tokens_enhanced, paper2_tokens_enhanced])
corpus_enhanced = [dictionary_enhanced.doc2bow(paper) for paper in [paper1_tokens_enhanced, paper2_tokens_enhanced]]

# Apply LDA
lda_model_enhanced = gensim.models.LdaModel(corpus_enhanced, num_topics=2, id2word=dictionary_enhanced, passes=15)
topics_enhanced = lda_model_enhanced.print_topics(num_words=5)
topics_enhanced

[(0, '0.017*"tree" + 0.014*"x" + 0.012*"1" + 0.011*"2" + 0.008*"branch"'),
 (1, '0.001*"tree" + 0.001*"x" + 0.001*"2" + 0.001*"1" + 0.001*"branch"')]

Great! It looks like the code executed successfully this time.

The LDA model has identified two topics from the provided papers. Here are the topics and the top 5 words associated with each:

Topic 0:
- tree
- V
- branch
- 1
- 2
Topic 1:
- (a special character)
- x
- (another special character)
- tree
- 1

From the topics, it seems that the word "tree" is prominent in both papers, which aligns with the context of Aronszajn trees that you mentioned earlier. However, there are also some special characters that have been identified as significant, which might be artifacts from the PDF extraction process or mathematical symbols.

The next steps would involve refining the preprocessing (e.g., handling special characters and mathematical symbols better) and interpreting the topics in the context of the papers.

Would you like to proceed with refining the preprocessing or move on to another step?

import pyLDAvis
import pyLDAvis.gensim_models as gensimvis

# Visualize the topics
pyLDAvis.enable_notebook()
vis = gensimvis.prepare(lda_model_enhanced, corpus_enhanced, dictionary_enhanced)
vis

Feature Engineering: Before feeding the data into a machine learning model, we need to convert the text data into a numerical format. We've already discussed using TF-IDF for this purpose. Another option is word embeddings like Word2Vec or FastText.
Model Selection: We discussed starting with simpler models like Multinomial Naive Bayes and then potentially exploring more complex models if needed.
Training: Once we've selected a model, we'll train it using our preprocessed data. Since we currently have only two papers, we might need more data for effective training.
Evaluation: After training, we'll evaluate the model's performance using appropriate metrics. For classification tasks, accuracy, precision, recall, and F1-score are commonly used metrics.
Optimization: Based on the evaluation results, we might need to optimize our model by tuning hyperparameters, using different algorithms, or gathering more data.
Deployment: Once we're satisfied with the model's performance, we can think about deploying it for real-world use. This could involve integrating it into a web application or other platforms.
Scaling Up: As we discussed, starting with simpler models and techniques gives us a foundation. Once we have this foundation, we can explore more advanced techniques, gather more data, and refine our approach.

Immediate Next Steps:

Convert the preprocessed text data into a numerical format using TF-IDF.
Split the data into training and testing sets (if we had more data).
Train a basic model (e.g., Multinomial Naive Bayes) on the training data.
Evaluate the model's performance on the testing data (if we had a split).

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

# Convert the preprocessed text data into a numerical format using TF-IDF
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform([' '.join(paper1_tokens_enhanced), ' '.join(paper2_tokens_enhanced)])

# For demonstration purposes, let's assume paper1 is labeled as '0' and paper2 as '1'
y = [0, 1]

# Train a basic Multinomial Naive Bayes model
model = MultinomialNB()
model.fit(X, y)

# Predict the class of the papers (just for demonstration)
predictions = model.predict(X)
predictions

array([0, 1])

from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier

# Create a dictionary to store the models and their predictions
models = {
    'MultinomialNB': MultinomialNB(),
    'LogisticRegression': LogisticRegression(),
    'SVC': SVC(),
    'RandomForest': RandomForestClassifier()
}
predictions = {}

# Train each model and store the predictions
for name, model in models.items():
    model.fit(X, y)
    predictions[name] = model.predict(X)

predictions

{'MultinomialNB': array([0, 1]),
 'LogisticRegression': array([0, 1]),
 'SVC': array([0, 1]),
 'RandomForest': array([0, 1])}

# List of major packages used in the project
major_packages = [
    'numpy',
    'pandas',
    'gensim',
    'nltk',
    'scikit-learn',
    'pyLDAvis',
    'PyPDF2'
]

# Extract versions of these packages
import pkg_resources
versions = {pkg: pkg_resources.get_distribution(pkg).version for pkg in major_packages}

# Write to requirements_major.txt
with open('requirements_major.txt', 'w') as f:
    for pkg, version in versions.items():
        f.write(f'{pkg}=={version}\n')

versions

{'numpy': '1.24.3',
 'pandas': '2.0.3',
 'gensim': '4.3.1',
 'nltk': '3.8.1',
 'scikit-learn': '1.3.0',
 'pyLDAvis': '3.4.1',
 'PyPDF2': '3.0.1'}